DOCUMENT RESUHE 



ED 352 523 CE 062 680 

TITLE Applied Workplace Literacy for the Facilities 

Maintenance Industry. Performance Report and Third 
Party Evaluation Report. 
INSTITUTION Texas State Technical Coll., Waco. 
SPONS AGENCY Office of Vocational and Adult Education (ED), 
Washington, DC. National Workplace Literacy 
Program. 
92 

V198A10073 
109p. 

Reports - Descriptive (141) 
MF01/PC05 Plus Postage. 

Adult Basic Education; ^Adult Literacy; Educational 
Innovation; Educational Technology; Facilities; 
Instructional Design; ^Instructional Material 
Evaluation; ^Interactive Video; Learning Modules; 
Maintenance; Material Development; ^"^Mathematics 
Instruction; *Mult imedia Instruction; Outcomes of 
Education; Programed Instruction; Program 
Effectiveness ; *Videodisks 
^Workplace Literacy 
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differences in results based on age or gender. The results of the 
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W 1 . Compare actual accomplishments to the objectives contained in the 

approved application. 

Introduction 

As shown in a variety of publications to include the Hudson Institute's 
Workplace 2000 and the SCANS Report, the United States faces a crises in the 
workplace, and one industry needing workers with basic literacy skills is the facilities 
maintenance business sector. Toxic chemicals, safety reports, and computerization 
issues all add to a facilities maintenance workplace that is rapidly changing and 
necessitates a skilled workforce to accommodate that change. Chemicals mixed in the 
wrong formula, measurements not correctly read, and geometric shape miscalculation 
contribute to lower productivity and increased service cost. With these problems 
noted, ServiceMaster, one of the largest facility maintenance companies in the world, 
partnered with Texas State Technical College Waco to develop a contextual-based 
videodisc curriculum for facility maintenance workers intended to increase math 



technology-enhanced curriculum and training through the jPgAS Center. Research and 
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literacy skills. 



Texas State Technical College Waco had been pioneering several applied 
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curriculum delivery methods through the Center for Applied Learning and developing 
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application found tfiat a comprehiensive interactive delivery system with the 
appropriate peripherals could provide the delivery strategy necessary to increase 
literacy scores for the facilities maintenance scores. The questions, then, to resolve in 
this study were: (a) Would a contextual multimedia training program increase math 
literacy levels for facilities maintenance personnel who most need help; and (b), would 
there be any differences regarding age and gender. The appropriate parties completed 
the partnership agreements, and the program began in April, 1991. 
Review of the Literature 

Instructional Technoloqv. Interactive multimedia has consistently shown to be a 
very diverse instructional technology medium (Bosco, 1986). When appropriately 
designed and implemented, test results have shown to be positive in comparison to 
traditional delivery systems when adults are the target audience (Hannafin & Colamaio, 
1987; Hannafin & Phillips, 1987; and Ziegler, 1990). 

Adult Literacy. Adult literacy programs have, in general, be3n delivered in 
traditional formats (Campbell & Sechler, 1987). Many studies (Long, 1980; Lotto, 
1983) have shown successful results in delivering contextual literacy training, yet, 
traditional methodologies have remained the delivery focus. According to Campbell 
and Sechler (1987), recommendations for research include: 

1 . Conduct systematic research of the basic skills as they apply to 
occupational of adult basic education. 

2. Identify and assess alternative delivery practices for adult basic 
literacy; 

This research agenda hints at two questions concerning all education: (a) what to 



teach and (b) how to teach It. This study, then, set forth to determine if interactive 
multimedia (how to teach) could produce positive results in literacy programs (what to 
teach) and answer of the questions set forth by the research agenda of Campbell and 
Sechler (1987). 
Methodoloqv 

Product Development. Staff from Texas State Technical College Waco 
developed a task analysis, curriculum, and evaluation meas jres for the training of 
facilities maintenance workers in math concepts. From this, the appropriate 
instructional design, workbook activitis^s, and scripting activities for the videodisc were 
developed. Synergistic Educational Technology Systems served as the production 
group for both the videodisc and software. Shot on location with Texas State 
Technical College Waco physical plant employees as actors, development time took 
four months for a usable training product. 

Audience. As shown in Figures 1 , 2, and 3, the total population reflected a 
diverse ethnic population in a variety of age categories with a variety of levels of 
education. The sample mirrored that population (Figures 4, 5, and 6). Using the 
SelectABLE as an assessment tool, the results suggested three distinct groups: (a) 0 - 
4th grade level; (b) 5th to 8th grade level; and (c), 9th and above. Thirty physical 
plant employees volunteered for training and participated based on their level as 
determined by the assessment. 

Implementation. Each group was further assessed using the ABLE test to 
establish a pretest score. From here, each participant received training from the 
videodisc. As this was an individualized program, each participant worked in a 



prescribed lesson plan six hours per week as dictated by the videodisc's pretest. 
Upon completion of each videodisc unit, the participants undertook accompanying 
workbook exercises for additional practice. Level 3 participants completed the training 
much more quickly than the Level 1 group as they originally could master more of the 
math competencies. Average completion time of the program was 60 hours. The 
ABLE test was administered to determine posttest scores. 

Variables. For this study, posttest scores (number operations and problem 
solving) on the ABLE test were used as the dependent variables. Independent 
variables were age and gender. The pretest served as the covariate. 

A list of the activities and objectives, with time lines, is included with the task 
chart. Ail of these have been completed. Each objective identified in the grant is 
addressed in the evaluation plan and was successfully completed within the timelines 
and budget identified in the original grant proposal application. 



2. Refer to the schedule of accomplishments and their target dates 
contained in the approved application and give reasons for slippage in 
those cases where established objectives not met. Include any 
corrective measures -aken to correct slippage. 

All the accomplishments were completed within the timelines specified. A delay 
in the original grant was caused by late notification by the Department of Education to 
TSTC Waco of the original award. This delay resulted in a change from the original 
proposal timelines of approximately 3 months slippage. This change in the timeline 
resulted in an conflict of summer work schedules , which are quite heavy, with the 
employees involved in the training. Due to the different timelines during the work 
year, the number of participants for the start-up training in the first quarter was 
significantly less than originally proposed. In addition to the revised schedule, budget 
and personnel cuts were enacted by ServiceMaster during the summer for cost cutting 
measures at the campus. These cuts reduced the number of employees and increased 
the workload of remaining employees. These budget cuts also affected the number of 
participants participating in the educational activities. The effect of the Jate start and 
th(5 limited availability of people to be trained resulted in the delay or slippage in 
testing of the interactive videodisc materials with a prototype training group. This 3 
month delay was made up by the developer and by TSTC during the second quarter. 
The project was on schedule with the training each quarter. The number of trained 
personeil was less than anticipated because of the yearly timing and workplace 
seasonal scheduling. This delay in completion of the training was about one and 1 12 



months of slippage. The training was completed by August 31, 1992 as specified bv 
the grant but the field testing of the completed interactive video materials was not 
completed until the end of September, 1992. Since a no-cost extension on the project 
of 30 days was asked for In order for the project director to attend a 1992 National 
Workplace Literacy Project Directors' Close-Out Conference on the grant, all the 
activities in the project were completed within the adjusted time-lines. 



3. For projects involving direct services to individuals identify the 
number and characteristics of project participants who completed 
planned project activities and of those who did not, and the outcomes 
achieved by participants who completed project activities. 

Data Analysis 

Determination of Effectiveness. Looking over the posttest scores {Figures 7, 8, 
and 9), one can see that program did bring many of the participants up in regards to 
mastery level in their assigned level. Further investigation into Levels 1 and 2 indicate 
these groups had the highest level of increase. Level 1 averaged a twenty-seven 
point increase in number operations and an eighteen point increase in problem solving. 
The Level 2 group averaged a twenty-eight point incjaase in number operations and a 
five point increase in problem solving. Therefore, using interactive video as a delivery 
medium with the appropriate contextual instructional design does assist in increasing 
math literacy levels. 

Gender and Age Differences. To determine if any gender or age differences 
existed, two 3x2 ANCOVAs were used with age and gender as the independent 
variables while the pretest served as covariate. Looking at number operations (Table 
1), there were no significant differences between the groups, F{1, 30) = .291, 
p<.05. Using the same ANCOVA structure with problem solving as the dependent 
variable (Table 2), there was once again no significant differences among the groups, 
F(1, 30) = .678, p<.05. No significant differences were found in either case. The 
videodisc performed the same for each participant regardless of age or gender. 



Table 1 

ANCOVA Summary Table for Hypotheses using Number Operations as the Dependent 
Variable. 



Source of 
Variation 



Sum of 
Squares 



DF 



Mean F 
Square 



Sig, 
of F 



Covarlates 



PRETEST 



235.S74 
235.674 



235.674 
235.674 



2.468 
2.468 



.130 
.130 



Main Effects 



Main Effects 

AGE 

SEX 



83.246 
81.059 
4.814 



3 
2 
1 



27.749 
40.529 
4.814 



.291 
.424 
.050 



.832 
.659 
.824 



Two-Way Interactions 



2-Way 
Interactions 
AGE X SEX 



238.884 
238.884 



2 

2 



119.442 
119.442 



1.251 
1.251 



.305 
.305 



Note. *a<.05. 



Table 2 

ANCOVA Summary Table for Hypotheses using Problem Solving as the Dependent 
Variable. 



Source of 
Variation 



Sum of 
Squares 



DF 



Mean 
Square 



Sig. 
of F 



Covariates 



Pretest 



5165.807 
5165.807 



5165.807 
5165.807 



26.307 
26.307 



.000 
.000 



Main Effects 



Main Effects 

AGE 

SEX 



399.297 
322.931 
39.158 



3 
2 
1 



133.099 
161.465 
39.158 



.678 
.822 
.199 



.575 
.452 
.659 



Two-Way Interactions 



2-Way 
Interactions 
AGE X SEX 



251.267 
251.267 



2 

2 



125.634 
125.634 



.640 
.640 



.537 
.537 



Note . *fi<.05. 
Figures 1 -9 Waco Site 



Further testing . TSTC Waco and ivy Tech cooperated on a pilot project to 
determine the effectiveness of the videodisc using just the videodisc pre and post 
tests as the evaluation instrument. This showed the reliability of the test to be .66 
using the Spearman-Brown prophecy formula, and the Kuder-Richardson 20 method 
determined Coefficient alpha to be .71. As reflected in the posttest scores(M = 82.5, 
SD = 4.6, n = 27) the average gain was 4.4 points. Those students who scored 
below 70 on the pretest scored above the 80 level on the posttest score. This 
indicated that the videodisc met its goal of achieving 80% mastery upon completion. 
Once again, student evaluation of the videodisc showed it to be helpful, energetic, and 
useful instruction for the workplace. 
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Discussion 

Conclusions. The results of the study showed that a contextual videodisc- 
based instructional delivery system yields positive results in adult literacy programs. 
Both Levels 1 and 2 improved their scores in number operations and problem solving. 
While scores increased, the training did not have any bias towards age and gender in 
regards to number operations (F(1, 30) = .291, p<.05) and problem solving scores 
(Fd, 30) = .678, p<.05). 

Implications and Recommendations . This study has direct implications for 
workplace literacy programs. Using the technology to deliver the literacy training 
allows workers to complete training regardless of the presence of an instructor. This 
means that training can take place twenty-four hours a day, and the trainer can spend 
more time developing curriculum as well as materials. Using the technology also 
enhances the opportunities for contextual-based learning versus the traditional system. 
This allows learners to become more involved with the learning as it reflects their 
everyday occupation rather with some distant environment with which they could not 
relate. This ownership of learning is possibly seen in the increase in posttest scores 
by Levels 1 and 2. 

In regards to recommendations, further study needs to be done in the use of 
technology in a variety of workplace literacy programs. Literacy issues may be a 
result of changing learning styles rather than disabled or slow learners. Testing 
technology in a variety of settings will help validate this process. One must 
remember, however, that it is the instructional design, not the hardware/software that 
dictates the success of the program. Secondly, continual testing of literacy programs 



using technology would enable more people to experience the changes in the 
workplace. This could drive increased academic as well as technical literacy as the 
United States tries to upgrade the skill of its workforce. In a global market with 
international implications for all business and industry, the United States must 
encourage efforts to increase literacy skills in order to achieve a dominant role in the 
world economy. 
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4. Report on any dissemination activities. 

The following presentations, tours and demonstrations were conducted during 
the project. 



January 22, 1992 


Project Advisory Committee Meeting 


January 24,1392 


Waco, Connally and La Vega Independent Schools 


February 3, 1 992 


Weyerhauser, Incorporated Tour 


February 4, 1 992 


Waco Independent School District Administration 


February 5, 1992 


ServiceMaster Corporate Training Staff 


February 1 2, 1 992 


Region Superintendents of Schools 


February 18, 1992 


Providence Hospital Training Staff 


February 20, 1 992 


Bowie High School Administration 


March 2. 1992 


Baylor University Educators 


March 4, 1992 


Waco Independent School District Instructors 


March 5, 1992 


Alliance Airframe, Dallas 



March 8, 9 and 11,1 992 National Tech-Prep Consortium Tours and Presentation 
March 1 2, 1 992 City of Austin 

Texas State Technical College System Board of Regents 
Tarrelton State University Instructors 
Texas Educational Region Service Center Staff, Region 1 3 
Marlin Independent School District Administration 
Shiilstone Inc., Dallas 

Texas Department of Transportation, Austin 
Huck Engineering, Inc. 



March 27, 1992 
April 10, 1992 
April 13, 1992 
April 14, 1992 
April 22, 1992 
April 24, 1992 
April 29, 1992 



4S , 



May 1 , 1 992 Presentation at Weyerhauser 

May 1 , 1 992 Project Advisory Committee Meeting 

May 11,1 992 Sedle Group 

May 24-27, 1 992 National Institute Staff and Organizational Development 

Conference, Austin 

June 10-17 and June 25- Julyl, 1992 Center for Occupational Research and 

Development Train-the-Trainer 
Workshops 

June 23-24, 1 992 Department of Transportation, Labor and Education 

Presentation, Washington, D.C. 
July 6, 1992 ServiceMaster Corporate Training Staff 

July 1 5, 1 992 Federal Department of Transportation, Fort Worth 

August 4, 1 992 Texas State Senate Subcommittee Hearing, Forth Worth 

August 1 1 - 14, 1 992 National Bridges Conference 
August 31, 1992 ALICO Life Insurance Company 

August 27, 1 992 Project Advisory Committee 

September 9- 11,1 992 Workplace Literacy Project Closeout Conference, 

Washington, D.C. 

September 1 7, 1 992 Texas State Council on Vocational Education Tour 
September 23, 1 992 North West Waco Lions Club 
October 2, 1 992 Texas Association of School Boards 

October 7, 1992 Marlin Rotary Club 



October 21- 24, 1992 Presentations at the League for Innovation in the Community 

Colleges 

December 4, 1 992 American Vocational Association presentation 
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5. Report on any evaluation activities. 

Background 



Evaluation of education efforts is a common, necessary requirement of all 
projects financed by State and Federal Agencies. Program or project evaluations have 
been used as a mechanism to identify which innovations succeeded and which did 
not. However, evaluation has not been an area of concentration by many individual 
educators. The nebulous nature of too many products of human learning and 
education activities has contributed to this lack of concentration. 

The need for developing and using an appropriate evaluation for the program or 
project to be evaluated has been indirectly emphasized by Alan Ginsburg (1992), who 
stated: 

One criticism was that evaluators showed a preoccupation with measuring 
overall program impacts, particularly test score changes, while achievement 
outcomes are important, they don't tell the whole story. "Black box" 
evaluations that ignore program processes are particularly frustrating in that, by 
themselves, they fail to indicate how to improve poorly performing programs. 
With these evaluation concerns in mind, staff and consultants of the Applied 
Workplace Literacy for the Facilities Maintenance Industry developed and used a 
project evaluation plan that combined three stages. The first state of the plan was to 
obtain formative information related to the installation of the project. The second 
stage dealt with process and was used to ensure that the activities of the project 
management plan would achieve the state project objectives. The third stage of the 



evaluation plan, in a summative mode, was designed to provide information on the 
project products and achievements. 

The goal of the demonstration project "Applied Workplace Literacy for the 
Facilities Maintenance Industry" was to develop a pilot workplace training program 
that could be a model for basic skills instruction for the majority of Service Master's 
thousands of support service employees worldwide. 

Each project objective was analyzed for the purpose of Identifying tasks or 
activities that were to be performed in the achievement of each of the four project 
objectives. Each of the tasks/activities identified by an analysis of each objective as 
categorized by each of the three stages of the evaluation. For each task/activity 
selected for review (evaluation), persons to be involved in the review of each 
task/activity were identified. Questions or criteria were developed for evaluating each 
task/activity performed and/or product produced. The responses of the reviewers of 
each task/activity and/or product are reported by evaluation stage in the following 
narrative. 

Presentation of Evaluation Data 

The evaluation of the project was conducted using a series of survey forms for 
the task/activities to be evaluated. Following, organized into three evaluation stages, 
are the responses to the questions on the survey forms. 



Installation Stage 



Task/Activity: Project Management Plan 



The persons involved in the review of this task/activity included three instructors, one 
partner, three project administrative staff members, one other project staff member, 
and the independent evaluator. Following are the responses of the reviewers for each 
question used for the evaluation of this task/activity. 



Questions 
1. 



Were individual tasks listed and distributed by project 
objective? 



2. Were milestones and timelines established by tasks and 
objectives? 

3. Did the management plan provide for modifications and 
revisions as needed? 

4. Was the management plan shared with all parties involved 
in the project? 



Percent of 
Reviewers 
Responding 

YesIQQ No_ 
YeslfiQ No_ 
Yesim Nc_ 
Yes 100 No 



Task/Activity: Evaluation Plan 

The persons involved in the review of this task/activity included two instructors, one 
partner, one other project staff member, two project administrative staff members, 
and the independent evaluator. Following are the responses of the reviewers for each 
question used for the evaluation of this task/activity. 



Questions 



Were you as a member of the project staff, participating 
partner, or Third party evaluator involved in the 
development of the evaluation plan? 

Do you feel that the evaluation plan provided for the 
collection of information that would include both 
qualitative and quantitative measure? 



Percent of 
Reviewers 
Responding 

Yes 100 No 



Yes 100 No 
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3. Did the evaluation plan include both formative and 
summative evaluation? 



4. Is the project being evaluated in accordance with the 
evaluation plan? 



Yes 100 No_ 
Yesim No_ 



Task/ Activity: Dissemmation Plan 

The persons involved in the review of this task/activity included three instructors, and 
four project administrative staff members. Following are the responses of the 
reviewers for each question used for the evaluation of this task/activity. 

Percent of 
Reviewers 
Responding 

Yes 100 No_ 

Yes 100 No_ 

Yes 100 No_ 



Questions 

1. Was the dissemination plan installed as proposed? 

2. Were each of the avenues (activities) for project 
dissemination described in the plan used? 

3. Are there indicators that the dissemination plan was 
effective? 



Task/Activity: Staff DeveSopment 

The persons involved in the review of this task/activity included four instructors and 
four project administrative staff members. Following are the responses of the 
reviewers for each question used for the evaluation of this task/activity. 



Questions 

1. Was a staff development plan in place? 

2. If a staff development plan was not in place, was one 
developed? 

3. Was the existing plan revised or a new plan developed to 
provide for staff development needs unique to the project 
staff? 



Percent of 
Reviewers 
Responding 

Yes 100 No_ 

Yes 100 No_ 

Yes 100 No _ 
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4. Has the staff development plan been Implemented for Yes 10Q No _ 

project staff? 



5. Does the staff development plan appear to be meeting the Yes 100 No _ 
professional development needs of the project staff? 

Task/ Activity: Needs Assessment 

The persons involved in the review of this task/activity included three instructors, one 
partner, two project administrative staff members, and the independent evaluator. 
Following are the responses of the reviewers for each question used for the evaluation 
of this task/activity. 

Percent of 
Reviewers 

Questions Responding 

1 . Do you feel that appropriate tools (test, interview, Yes 100 No _ 
counseling) were used to determine the training needs of 

the students? 

2. Were the tools used appropriately for determining the 
training needs of students? 

3. Do you feel that appropriate tools were used to determine 
the level of instruction that should be provided for each 
student? 

4. Do you feel that the results of the training needs 
assessment were used in the planning and the delivery of 
instruction? 

Task/ Activity: Test Selection 

The persons involved in the review of this task/activity included two instructors, one 
partner, and two project administrative staff members. Following are the responses of 
the reviewers for each question used for the evaluation of this task/activity. 



Yes 100 No 



Yes 100 No_ 



Yes 100 No 



Questions 

1 . Were tests selected to meet the proposed purpose of 
testing? 



Percent of 
Reviewers 
Responding 

YeslIJQ No 
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Process Stage 



Task/Activity: Literacy Audit 

The persons involved in the review of this task/activity included five instructors, four 
project advisory committee members, three project administrative staff members, one 
partner, and twenty-three students. Following are the responses of the reviewers for 
each questions used for the evaluation of this task/activity. 



Questions 

1 . Was the literacy audit conducted as proposed? 

2. Did the literacy audit produce the products (results) 
proposed for the project? 

3. Were the findings of the literacy audit sufficient to 
develop a competency-based curriculumts)? 

4. Did the literacy audit identify both competencies and tasks 
of required basic skills? 



Percent of 
Reviewers 
Responding 

YeslOQ No_ 

Yes 100 No_ 

. Yes IQO No _ 

Yes 100 No _ 



Task/ Activity: Test Selection 

The persons involved in the review of this task/activity included two instructors, one 
partner, and two project administrative staff members. Following are the responses of 
::he reviewers for each question used for the evaluation of this task/activity. 



Questions 
1. 



2. 
3. 



Were the tests selected capable of determining the 
anticipated increase in worker's reading/ 
writing/mathematics skill levels? 

Were the test administered as proposed? 

Were the results of the tests used as proposed? 



Percent of 
Reviewers 
Responding 

YeslQQ No_ 



Yes 100 No 
Yes 100 No 



4. Were standardized tests used? 



Yes 100 No 



Task/Activity: Task Analysis 



The persons involved in the review of this task/activity included two instructors, two 
project administrative staff members, and the independent evaluator. Following are 
the responses of the reviewers for each question used for the evaluation of this 
task/activity- 



Questions 
1. 



Was an accepted occupational analysis procedure used to 
identify the tasks of each competency identified in the 
literacy audit? 



2. Was a proven task analysis procedure used? 

3. Was an analysis made of ail identified tasks? 

4. Were instructional topics (knowledge and performance) 
identified by each task analysis? 

5. Were the results of the task analysis compared with 
existing curricula and instructional content at TSTC 
Waco? 



Percent of 
Reviewers 
Responding 

Yes 100 No_ 



Yes 100 No_ 
YeslfiQ No_ 

Yes 100 No_ 
Yes 100 No_ 



Task/Activity: Program Design and Planning 

The persons involved in the review of this task/activity included two instructors, four 
project advisory committee members, and two project administrative staff members. 
Following are the responses of the reviewers for each question used for the evaluation 
of the task/activity- 

Percent of 
Reviewers 

Quest'ons Responding 

1 . Was the program designed and planned as described in Yes 100 No — 
the proposal? 

2. Were the appropriate personnel used in designing the Yes IfiQ No — 
program as proposed? 
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3. Were the proposed factors such as logistics and identified Yes 10Q No 

needs used in the design of the program? 



Task/Activity: Curriculum Development 

The persons involved in the review of this task/activity included three instructors, four 
project advisory committee members, two project administrative staff members, one 
contractor (SETS), and the independent evaluator. Following are the responses of the 
reviewers for each question used for the evaluation of this task/activity. 



Percent of 
Reviewers 

Questions Responding 

1. Were the curricula developed using the results of the Yes 100 No — 
literacy audit? 

2. Was the instructional content of the curricula developed Yes 100 No _ 
based on the results of the analysis of each identified 

task? 



3. Were objectives of existing instructional content modified Yes 100 No — 
to meet the instructional content required by the literacy 
audit? 



4. Are the curricula competency-based? Yes 100 No — 

5. Were individual differences in learning provided for in the Yes 100 No — 
instructional materials? 

6. Were appropriate methods for the delivery of instruction Yes 100 No _ 
provided for each lesson? 

7. Was sufficient time provided to allow for skills Yes 100 No _ 
improvement in the training schedule? 



Task/Activity: Instructional Materials (Purchased) 

The persons involved in the review of this task/activity included three instructors, one 
partner, one contractor, and two project administrative staff members. Following are 
the responses of the reviewers for each question used for the evaluation of this 
task/activity. 
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Questions 



Percent of 
Reviewers 
Responding 



1 . Were appropriate and sufficient instructional materials 
obtained that would implement the curriculum? 

2. Were modifications of the materials made when 
necessary? 

3. Were the materials field tested with students? 



YeslOQ No. 
YeslQO No 
Yes 100 No 



Task/Activity: Hardware and Software Selection 

The persons involved in the review of this task/activity included two instructors, one 
other project staff member, one contractor, and four project administrative staff 
members. Following are the responses of the reviewers for each question used for the 
evaluation of this task/activity. 

Percent of 
Reviewers 

Questions Responding 

1 . Were the evaluation procedures and standards developed Yes 100 No — 
for the selection of hardware and software based on the 

proposed methods of delivery of instruction described in 
the program design? 

2. Did the selected hardware and software meet the 
requirements of the delivery methods described in the 
program design? 

3. Did the software and hardware provide for the individual 
learning styles of trainees? 



Yes 100 No_ 



Yes 100 No 



Task/Activity: Student Training 

Thfe persons involved in the review of this task/activity included three instructors and 
two project administrative staff members. Following are the responses of the 
reviewers for each question used for the evaluation of this task/activity. 



Questions 



Percent of 
Reviewers 
ResDondinc 
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1. 



3. 
4. 

5. 
6. 
7. 
8. 



Were individual training plans developed for each 
student? 

Did the individual training plans reflect the results of the 
needs assessment for the respective student? 

Did the training proceed as scheduled? 

Did the majority of the students achieve the training 
(instructional) objectives? 

Was the training evaluated by staff and students? 
Was the time established for training sufficient? 
Were sufficient resources made available for training? 
Was student release time for training sufficient? 



Yes 92 No 8 

Yes 100 No _ 

Yes_92 No 8 
Yes 100 No _ 

Yes 100 No _ 
Yes 100 No _ 
Yes 100 No_ 
Yes 76 No 14 



Student Training was also evaluated by some of the students. Following are the 
responses of students who were asked to evaluate the training. 



Fvaluation Question 

1 . The amount of time allowed for this course was: 

2. The amount of work assigned was: 

3. The coursework applies to my job: 

4. I wish to participate in another class: 

5. The time of day for this class was: 



Number of students 
res ponding, by rating 

Too little _i 



About right 94 
Too much 0 



94 



0 



19 



62 



19 



Too little 
About right 
Too much 

Not at all 
Somewhat 
Frequently 

Never 
Later 
Immediately 53 

Too early Q_ 

About right 94 
Too late 6_ 



0 



47 



b J 



I would like more training in: 



25 



31. 



Reading 
Writing 
Math/Physics 
Computer 94 



44 



Task/ Activity: Selection of a Qualified Publisher/Contractor 

The persons involved in the review of this task/activity included three instructors, one 
contractor, and four project administrative staff members. Following are the 
responses of the reviewers for each question used for the evaluation of this 
task/activity. 



Questions 
1. 



Were capability and bid evaluation criteria 
established? 



2. Was a bid proposal format developed and used? 

3. Were state guidelines and bid procedures followed? 

4. Was a bid proposal evaluation form developed and used? 

5. Was the response to the bid proposals done in a timely 
manner? 



Percent of 
Reviewers 
Responding 

Yes IQQ No _ 



Yesl^ No_ 
Yes 100 No_ 
Yes 100 No_ 
YeslCJQ No_ 



Product Stage 



Task/Activity: Development of Interactive Video Disc (IVD) 

The persons involved in the review of this task/activity included three instructors, 
three project advisory committee members, three project administrative staff 
members, and one contractor. Following are the responses of the reviewers for each 
question used for the evaluation of this task/activity. 

Percent of 
Reviewers 

Questions Responding 
1 . Was the IVD developed as proposed? Yes ^ No IQ 



2. Did the IVD meet contract specifications? Yes _80 No 20 

3. Does the IVD provide for the development of all skills Yes _80 No 10 
identified by the literacy audit and the task analysis? 

Task/Activity: Interactive Videodisc Training 

The persons involved in the review of this task/activity included three instructors and 
one project admijiistrative staff member. Following are the responses of the reviewers 
for each question used for the evaluation of this task/activity. 



Questions 

1 . Did the IVD training proceed as scheduled? 

2. Did the IVD training meet the instructional objectives as 
determined by the literacy audit? 

3, Was the IVD training evaluated by students and staff? 

4, Was the time established for IVD training sufficient? 



Percent of 
Reviewers 
Responding 

YesJTB No 25 

Yes 100 No_ 

Yes 100 No_ 
YesIM No_ 



5. 



Were sufficient resources needed to supplement the IVD Yes 100 No _ 

training made available? 



Task/Activity: Project Management Plan 



The persons involved In the review of this task/activity included three instructors, one 
partner, four project administrative staff members, one contractor, and the 
independent evaluator. Following are the responses of the reviewers for each question 
used for the evaluation of this task/activity. 



Questions 



Percent of 
Reviewers 
Responding 



1 . Were the milestones and timelines met as established? Yes ^ No H 

2. Were the revisions made as the result of unforeseen barriers Yes 100 No — 

Task/ Activity: Program Design and Planning 

The persons involved in the review of this task/activity included two instructors, four 
project advisory committee members, and two project administrative staff members. 
Following are the responses of the reviewers for each question used for the evaluation 
of this task/activity. 



Questions 

1. Were the proposed resources used in the design? 
Were additional resources identified? 
Were the total resources sufficient? 



2. 
3. 
4. 



Were provision made in the design to assure the curricula 
would be competency-based? 



Percent of 
Reviewers 
Responding 

YeslIJQ No. 

YeslfiQ No 

Yes 100 No 

Yes 100 No 



5. Were the proposed methods of delivery used in the design? Yes 100 No 



Task/Activity: Student Achievement 

The persons involved in the review of this task/activity consisted of one partner. 
Following are the responses of the reviewers for each question used for the evaluation 
of this task/activity. 



Percent of 
Reviewers 

Questions Responding 

1. Rate the increase in the productivity of all employees 

as a group after training by checking one of the statements below 
which best indicates your rating of increased productivity: 

1 . No increase in productivity 

2. Some increase in productivity 

3. Moderate increase in productivity 10Q 

4 . Above average increase in productivity 

5. High increase in productivity 

2. Was there improvement in communications and mathematical Yes 100 No 
skills of employees? 

3. Was there improvement in the workplace attitude of Yes 100 No 
employees? 

4. Was there a decrease in absenteeism of employees? Yes 100 No 

5. Would you recommend that other employees needing literacy Yes 100 No 
improvement be provided this training? 
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Task/Activity: Students with iVD Training 



The persons Involved in the review of this task/activity included five partners. 
Following are the responses of the reviewers for each question for the evaluation of 
this task/activity. 



Percent of 
Reviewers 

Questions Responding 



1 . Rate the increase in the productivity of all employees as a 

group after training by checking one of the statements below 
which best indicates your rating of increased productivity: 

1 . No increase in productivity 40 



2. Some increase in productivity 

3. Moderate increase in productivity 40 

4. Above average increase in productivity 20 

5. High increase in productivity 

2. Was there improvement in communications and mathematical Yes 100 No 

skills of employees? 

3. Was there improvement in the workplace attitude of Yes 100 No — 
employees? 

4. Was there a decrease in absenteeism of employees? Yes No 80 



5. Would you recommend that other employees needing literacy Yes 100 No _ 
improvement be provided this training? 

6. Do you feel that students (employees) participating in Yes 100 No _ 
IVD training made greater progress in training using 

traditional methods of teaching? 



Task/Activity: Evaluation Plan IVD Field Test 

The persons involved in the review of this task/activity included instructors, project 
advisory committee members, project administrative staff and the independent 



evaiuator. Following are the responses of the reviewers for each question used for the 
evaluation of this task/activity. 



Percent of 
Reviewers 

Questions Responding 

1 . Was the field test conducted as proposed? Yes^ No_67_ 

2. Were the evaluations of the IVD by employees 

(students) positive? YesJLQO No 

3. Were the positive evaluations of IVD by students at the 

field test sites documented? YesJLQQ No 

4. Were the assessments of IVD made by the adult 

education specialists at the field test sites positive? Yes 100 No 

5. Did ServiceMaster supervisors at each site feel that the 

IVD would be applicable for their staff? Yes 100 No 

6. Did the IVD meet the specific objectives of the 

instructional program? YesJLQQ No 



7. Did the training meet the intent of the grant? Yes_100 No 



8. Did the training meet the training needs identified by the 

literacy audit? YesJOQ No_ 



Additional nnmments: The field test was delayed because of the delav in the pilot test 
at TSTC Waco. The field test was conducted with 29 employees at one site instead 
of 10 sites with 10 employees. This difference in number of sit es and employees 
resulted because ServiceMaster failed to identify additi onal sites for field testing. SETS 
was responsible for the field site testing activities. 



Summary of Evaluation Findings 

The summary of the evaluation findings are reported by task/activity In relation 
to the project objective for which the task/activity was performed. 

Objective 1 : "Conduct a thorough needs assessment/literacy audit on which to 
base the design of the program. " 

Task/Activity: Selection of Literacy Tests 

Five persons were selected to respond to the questions used In the evaluation of 
this task. All five of the respondents agreed that this task was performed as 
proposed. 



Task/ Activity: Literacy Audit 

Thirty-six persons which included twenty-three students responded to the four 
evaluation questions for this task/activity. All thirty-six respondents agreed that the 
literacy audit was conducted as proposed and achieved the purposes for which the 
literacy audit was performed. 



Objective 2: "Conduct Phase I Training' 



Task/Activity: Task Analysis 
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Two instructors, two project administrative staff, and the independent evaluator 
reviewed both the process and products of the task analysis. Ail reviewers agreed 
that the task analysis was conducted as proposed and the expected results were 
produced. 

Task/Activity: Program DesSgn and Planning 

The eight reviewers of this task, which included four members of the project 
advisory committee, ail agreed that the criteria for evaluating this task had been 
achieved. This task/activity was also evaluated under the Product stage of the 
evaluation process. 

Task/Activity: Curriculum Development 

Eleven persons reviewed both the process and product of this task. It was the 
consensus of the reviewers that all seven of the criteria used to measure the 
achievement of this task had been met. 

Task/Activity: Purchase of Instructional Materials 

Seven persons, including one partner, responded to the questions used in 
determining if this task had been performed as proposed. All of the reviewers 
responded "yes" to questions used in evaluating this task. 

Task/Activity: Student Training (phase I) 



The reviewers of this task/activity did not agree that ail of the criteria used in 
evaluating this task/activity had been met at 100 percent. One reviewer felt that 
based on his/her definition of "training plan" that individual training plans had not been 
developed. One reviewer felt that the training did not proceed as scheduled. Two of 
the respondent? did not agree that student release time was sufficient. Comment 
made by these two reviewers were: "The students should have been allowed to 
attend training without having to donate their personal time i.e., lunch hour," and 
"Time was supposed to be given by Servicemaster, not 1 for 1." The reviewers 
agreed that all of the other criteria used in evaluating this task had been met. 

The evaluations of the students for i-tudent training were positive. Fifty-three 
percent of the students responding to the questionnaire indicated that they desired to 
participate in additional training immediately. Although all of the students responding 
expressed a desire to participate in future training, two students did not feel that the 
coursework applied to their jobs. 

Comments made by the responding students included: 
"I really enjoyed the studies that we had. Thanks for all of your help." 
"I greatly enjoyed the classes and our teach*^ ' was an absolute delight. I would 
love to take more classes. Thank you." 

"I would like more time with computer operations." 

"I enjoyed the training very much. My instructor, Melinda Ziegler, made me 
very comfortable in the classroom." 

"I feel that the classes would be valuable to anyone willing to take them." 



Task/Activity: Selection of a Qualified/Publisher Contractor 
All of the individuals evaluating this task agreed that all of the evaluation criteria 
had been met. 

Task/Activity: Development of Video Disc (IVD) 

This task was evaluated by six project staff members, three project advisory 
committee members, and one staff member of the contractor. The evaluations of this 
task were performed prior to June 10, 1992. The evaluators of this task did not agree 
that the interactive video disc for the curriculum had been developed as proposed and 
within the timeframe established. Following are the comments made by evaluators 
who did not feel the task had been achieved as proposed: 

"IVD not the only thing to address deficiencies-other methods used to address 
same needs identified by the literacy audit." 

"At this time modifications to the IVD had not been received." 

"IVD has yet to be completed." 

Task/Activity: . Interactive Videodisc Training 
Three instructors and one project administration staff member reviewed the 
interactive videodisc training. Overall the reviewers agreed that the training was 
conducted as proposed. One evaluator did not agree that the training proceeded as 
scheduled. The comment of this evaluator was: "Due to late start training was 
delayed but progressed at projected rate." 
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Task/Activity: Program Design and Planning 

Part of the program design and training evaluation data was presented under the 
evaluation stage of installation. The remaining part of the evaluation of this 
task/activity was performed under the product stage. The evaluators of the product 
stage for Program Design and Planning agreed that the produce criteria had been met. 

Task/Activity: Student Achievement 

Student achievement was evaluated on-site by five ServiceMaster supervisors 
of the student/employees and by the use of ABLE pretest and posttest. 

Student achievement of Phase I training was evaluated by one supervisor 
(partner). The evaluator felt that there was a moderate increase in productivity as a 
result of the Phase I training. The evaluators' ratings were positive and would 
recommend that other employees needing literacy improvement be provided with the 
training. 

Five of the supervisors evaluated student achievement following the IVD 
training. One of the supervisors felt that the student-employees as a group achieved 
an above average increase in productivity as a result on the training. 

Two felt that there was not an increase in training and two felt that the student 
employees had made a moderate increase in productivity. One of the supervisors felt 
that the training had decreased the absenteeism of the student-employees. 

All of the supervisors agreed that there was an Improvement of the student- 
employees in the areas of communications and mathematical skills, and workplace 



attitude as a result of the training. 

Ail of the supervisors agreed that they would recommend the training for other 
employees needing literacy improvement. All of the supervisors agreed that student- 
employees participating in the IVD training made greater progress than those in 
training using traditional methods of teaching. 

Objective 3: "Addressing the Computational Literacy Needs at the Management 

Level" 

This objective was removed from the project during contract negotiations with 
the Department of Education. 

Objective 4: "Conduct a complete evaluation of the project and, if results are as 
expected, prepare for dissemination of the program throughout the corporation and 
other educational institutions interested in serving ServiceMaster workers nationally 
and internationally." 

Task/Activity: Oversight Committee 

This task/activity was evaluated by the independent evaluator. It was found 
that the oversight committee, which is called the Project Transformation Committee, 
served the project as proposed. The independent evaluator attended all of the 
Oversight Committee meetings. 
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Task/ Activity: Third-Party Evaluator 

The third-party evaluator, referred to as the independent evaluator, was 
obtained as proposed. The third-party evaluator assisted the project staff in 
developing and implementing a project evaluation plan in accordance with the 
specifications agreed to in the contract negotiations with the Department of 
Education. The evaluation plan developed to guide in the evaluation of the project 
was reviewed and approved by the Oversight Committee. 

Task/Activity: Field Test of the Interactive Videodisc 

The field test was a joint activity between SETS, the contractor, TSTC and 
ServiceMaster. SETS had, as part of it's contract, the field test support, location and 
trial was coordinated between sites suggested by ServiceMaster and SETS with 
support from TSTC Waco. Out of 7 sites suggested by SETS only one was able to 
field test within the time frame of the grant. At this site, 29 people went through the 
field testing. The field test was completed and the successful results were reported 
to TSTC Waco. 

Task/Activity: Management Plan 

The management plan was initially set in place at the beginning of the grant. 
Reports of the progress of the grant were sent to all parties regularly as the activities 
progressed. Grant timelines and activities were addressed in these periodic reports. 
All parties were informed by the grant Director of progress and completion within the 
grant and the management plan was adhered to and followed. 
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Conclusion 

Based on the observations made by the independent evaluator of the evaluation 
data collected, the project has achieved the goal of the demonstration project by 
developing, testing by implementation, and proven by student achievement a model 
program of applied Workplace Literacy. This model or demonstration program can be 
used throughout the nation by ServiceMaster, Inc. and adapted or replicated for use by 
other businesses and industries providing similar services and products. 

It is conclusion of the independent evaluator and of the project manger that the 
project has achieved ail of the objectives and tasks of the proposed project. 
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6. Report on any changes in key personnel. 

Personel changes involved only instructors, who were not key personnel, for the 
training. Qualified instructors were used throughout the training program. Personnel 
changes also took place at ServiceMaster during the grant. ServiceMaster maintained 
their responsiblities to the grant. 
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EVALUATION OF THE 
APPLIED WORKPLACE LITERACY DEMONSTRATION PROJECT 

FOR THE 

FACILITIES MAINTENANCE INDUSTRY 
Report of Third Party Evaluator 

Background 

Evaluation of education efforts is a common, necessary requirement of all projects 
financed by State and Federal Agencies. Program or project evaluations have been used 
as a mechanism to identify which innovations succeeded and which did not. However, 
evaluation has not been an area of concentration by many individual educators. The 
nebulous nature of too many products of human learning and education activities has 
contributed to this lack of concentration. 

The need for developing and using an appropriate evaluation for the program or 
project to be evaluated has been indirectly emphasized by Alan Ginsburg, who stated: 
One criticism was that evaluators showed a preoccupation with measuring 
overall program impacts, particularly test score changes, while achievement 
outcomes are important, they don't tell the whole story. ''Black box'' 
evaluations that ignore program processes are particularly frustrating in that, by 
themselves, they fail to indicate how to improve poorly performing programs} 
With these evaluation concerns in mind, staff and consultants of the Applied Workplace 
Literacy for the Facilities Maintenance Industry developed and used a project evaluation 
plan that combined three stages. The first stage of the plan was to obtain formative 

^Ginsburg, AJan et al, Reinvigorating Program Evaluation at the U.S. Department of 
Education, EDUCATIONAL RESEARCHER, Vol. 21, Number 3, April 1992 
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information related to the installation of the project. The second stage dealt with 
process and was used to ensure that the activities of the project management plan would 
achieve the stated project objectives. The third stage of the evaluation plan, in a 
summative mode, was designed to provide information on the project products and 
achievements. 

The goal of the demonstration project "Applied Workplace Literacy for the 
Facilities Maintenance Industry" was to develop a pilot workplace training program that 
could be a model for basic skills instruction for the majority of Service Master's 
thousands of support service employees worldwide. 

Each project objective was analyzed for the purpose of identifying tasks or 
activities that were to be performed in the achievement of each of the four project 
objectives. Each of the tasks/activities identified by an analysis of each objective was 
categorized by each of the three stages of the evaluation. For each tas.'</activity selected 
for review (evaluation), persons to be involved in the review of each task/activity were 
identified. Questions or criteria were developed for evaluating each task/activity 
performed and/or product produced. The responses of the reviewers of each 
task/activity and/or product are reported by evaluation stage in the following narrative. 

Presentation of Evaluation Data 

The evaluation of the project was conducted using a series of survey forms for the 
task/activities to be evaluated. Following, organized into three evaluation stages, are the 
responses to the questions on the survey forms. 
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Installation Stage 



Task/Activity: Project Management Plan 

The persons involved in the review of this task/activity included three instructors, one 
partner, three project administrative staff members, one other project staff member, and 
the independent evaluator. Following are the responses of the reviewers for each 
question used for the evaluation of this task/activity. 



Questions 

1. Were individual tasks listed and distributed by project 
objective? 

2. Were milestones and timelines established by tasks and 
objectives? 

3. Did the management plan provide for modifications and 
revisions as needed? 

4. Was the management plan shared with all parties involved 
in the project? 



Percent of 
Reviewers 
Responding 

Yes 100 No 

Yes 100 No 

Yes 100 No 

Yes 100 No 



Task/Activity: Evaluation Plan 

The persons involved in the review of this task/activity included two instructors, one 
partner, one other project staff member, two project administrative staff member, and 
the independent evaluator. Following are the responses of the reviewers for each 
question used for the evaluation of this task/ activity. 

Percent of 
Reviewers 

Questions Responding 



Were you as a member of the project staff, participating 
partner, or Third party evaluator involved in the 
development of the evaluation plan? 



Yes 100 No 
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Installation Stage 



2. Do you feel that the evaluation plan provided for the Yes lOQ No . 
collection of information that would include both 

qualitative and quantitative measure? 

3. Did the evaluation plan include both formative and Yes 100 No 
summative evaluation? 

4. Is the project being evaluated in accordance with the Yes 100 No 
evaluation plan? 



Task/Activity: Dissemination Plan 

The persons involved in the review of this task/activity included three instructors, and 
four project administrative staff members. Following are the responses of the reviewers 
for each question used for the evaluation of this task/activity. 

Percent of 
Reviewers 

Questions Responding 

1. Was the dissemination plan installed as proposed? Yes 100 No 

2. Were each of the avenues (activities) for project Yes 100 No 

dissemination described in the plan used? 

3. Are there indicators that the dissemination plan was Yes 100 No 

effective? 



Task/Activity: Staff Development 

The persons involved in the review of this task/activity included four instructors and four 
project administrative staff members. Following are the responses of the reviewers for 
each question used for the evaluation of this task/activity. 

Percent of 
Reviewers 

Questions Responding 

1. Was a staff development plan in place? Yes 100 No 
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2. If a staff development plan was not in place, was one 
developed? 

3. Was the existing plan revised or a new plan developed to 
provide for staff development needs unique to the project 
staff? 

4. Has the staff development plan been implemented for 
project staff? 

5. Does the staff development plan appear to be meeting the 
professional development needs of the project staff? 



Installation Stage 
Yes 100 No 

Yes 100 No 

Yes 100 No 

Yes lOQ No 



Task/Activity: Needs Assessment 

The persons involved in the review of this task/activity included three instructors, one 
partner, two project administrative staff members, and the independent evaluator. 
Following are the responses of the reviewers for each question used for the evaluation of 
this task/activity. 



Questions 

1. Do you feel that appropriate tools (tests, interview, 
counseling) were used to determine the training needs of 
the students? 

2. Were the tools used appropriately for determining the 
training needs of students? 

3. Do you feel that appropriate tools were used to determine 
the level of instruction that should be provided for each 
student? 

4. Do you feel that the results of the training needs 
assessment were used in the planning and the delivery of 
instruction? 



Percent of 
Reviewers 
Responding 

YeslOQ No 



Yes lOQ No . 
YeslOQ No 

Yes 100 No 



Task/Activity: Test Selection 
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Installation Stage 



The persons involved in the review of this task/activity included two instructors, one 
partner, and two project administrative staff members. Following are the responses of 
the reviewers for each question used for the evaluation of this task/activity. 

Percent of 
Reviewers 

Questions Responding 

1. Were tests selected to meet the proposed purpose of Yes 100 No 

testing? 



Process Stage 



Task/Activity: Literacy Audit 

The persons involved in the review of this task/activity included five instructors, four 
project advisory committee members, three project administrative staff members, one 
partner, and twenty-three students. Following are the responses of the reviewers for 
each question used for the evaluation of this task/activity. 



Questions 

1. Was the literacy audit conducted as proposed? 

2. Did the literacy audit produce the products (results) 
proposed for the project? 

3. Were the findings of the literacy audit sufficient to 
develop a competency-based curriculum(s)? 

4. Did the literacy audit identify both competencies and tasks 
of required basic skills? 



Percent of 
Reviewers 
Responding 

Yes 100 No. 

Yes 100 No . 

Yes 100 No 

Yes 100 No 



Task/Activity: Test Selection 
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Process Stage 



The persons involved in the review of this task/activity included two instructors, one 
partner, and two project administrative staff members. Following are the responses of 
the reviewers for each question used for the evaluation of this task/activity. 



Questions 

1. Were the tests selected capable of determining the 
anticipated increase in worker's reading/ 
writing/mathematics skill levels? 

2. Were the tests administered as proposed? 

3. Were the results of the tests used as proposed? 

4. Were standardized tests used? 



Percent of 
Reviewers 
Responding 

Yes 100 No 



Yes 100 No 

Yes 100 No 

Yes 100 No 



Task/Activity: Task Analysis 

The persons involved in the review of this task/activity included two instructors, two 
project administrative staff members, and the independent evaluator. Following are the 
responses of the reviewers for each question used for the evaluation of this task/activity. 



Percent of 
Reviewers 

Questions Responding 



1. Was an accepted occupational analysis procedure used to Yes 100 No 

identify the tasks of each competency identified in the 

literacy audit? 

2. Was a proven task analysis procedure used? Yes 100 No 

3. Was an analysis made of all identified tasks? Yes IQO No 

4. Were instructional topics (knowledge and performance) Yes lOQ No 

identified by each task analysis? 
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Process Stage 

5. Were the results of the task analysis compared with Yes 100 No 

existing curricula and instructional content at TSTC 
Waco? 



Task/Activity: Program Design and Planning 

The persons involved in the review of this task/activity included two instructors, four 
project advisory committee members, and two project administrative staff members. 
Following are the responses of the reviewers for each question used for the evaluation of 
this task/activity. 



Questions 



Percent of 
Reviewers 
Responding 



1. Was the program designed and planned as described in 
the proposal? 

2. Were the appropriate personnel used in designing the 
program as proposed? 



Yes 100 No 



Yes 100 No 



3. Were the proposed factors such as logistics and identified 
needs used in the design of the program? 



Yes 100 No 



Task/Activity: Curriculum Development 

The persons involved in the review of this task/activity included three instructors, four 
project advisory committee members, two project administrative staff members, one 
contractor (SETS), and the independent evaluator. Following are the responses of the 
reviewers for each question used for the evaluation of this task/activity. 

Percent of 
Reviewers 

Questions Responding 

1. Were the curricula developed using the results of the Yes lOQ No 

literacy audit? 
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2. Was the instructional content of the curricula developed Yes IQQ No . 
based on the results of the analysis of each identified 

task? 

3. Were objectives of existing instructional content modified Yes 100 No . 
to meet the instructional content required by the literacy 

audit? 

4. Are the curricula competency-based? Yes 100 No , 

5. Were individual differences in learning provided for in the Yes 100 No . 
instructional materials? 

6. Were appropriate methods for the delivery of instruction Yes 100 No 
provided for each lesson? 

7. Was sufficient time provided to allow for skills Yes 100 No 
improvement in the training schedule? 



Task/Activity: Instructional Materials (Purchased) 

The persons involved in the review of this task/activity included three instmctors, one 
partner, one contractor, and two project administrative staff members. Following are the 
responses of the reviewers for each question used for the evaluation of this task/activity. 

Percent of 
Reviewers 

Questions Responding 

1. Were appropriate and sufficient instructional materials Yes 100 No 

obtained that would implement the curriculum? 

2. Were modifications of the material made when Yes 100 No 

necessary? 

3. Were the materials field tested with students? Yes IQO No 



Task/Activity: Hardware and Software Selection 



Process Stage 



The persons involved in the review of this task/activity included two instructors, one 
other project staff member, one contractor, and four project administrative staff 
members. Following are the responses of the reviewers for each question used for the 
evaluation of this task/activity. 



Questions 

1. Were the evaluation procedures and standards developed 
for the selection of hardware and software based on the 
proposed methods of deliveiy of instruction described in 
the program design? 

2. Did the selected hardware and software meet the 
requirements of the delivery methods described in the 
program design? 

3. Did the software and hardware provide for the individual 
learning styles of trainees? 



Percent of 
Reviewers 
Respondin g 

Yes 100 No 



Yes 100 No 



Yes 100 No 



Task/Activity: Student Training 

The persons involved in the review of this task/activity included three instructors and 
two project administrative staff members. Following are the responses of the reviewers 
for each question used for the evaluation of this task/activity. 



Questions 

1. Were individual training plans developed for each 
student? 

2. Did the individual training plans reflect the results of the 
needs assessment for the respective student? 

3. Did the training proceed as scheduled? 

4. Did the majority of the students achieve the training 
(instructional) objectives? 



Percent of 
Reviewers 
Responding 

Yes 92 No 8 



Yes 100 No_ 

YesJ2^ No_8. 
Yes 100 No _ 
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5. Was the training evaluated by staff and students? Yes 100 No 

6. Was the time established for training sufficient? Yes IQO No 

7. Were sufficient resources made available for training? Yes 100 No 

8. Was student release time for training sufficient? Yes 76 No ^16_ 
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Student Training was also evaluated by some of the students. Following are the 
responses of students who were asked to evaluate the training. 

Number of students 

Evaluation Question responding, by rating 

1. The amount of time allowed for this course was: Too little _L 

About right 15 
Too much _0L 



2. The amount of work assigned was: Too little _L 

About right J5 
Too much 0 



3. TY^e, coursework applies to my job: Not at all 

Somewhat 
Frequently 



4. I wish to participate in another class: Never _0_ 

Later 8 
Immediately 



5. The time of day for this class was: Too early _0_ 

About right 15 
Too late _L 



6. I would like more training in; Reading _4_ 

Writing 5 
Math/Physics 7 
Computer 15 



Task/Activity: Selection of a Qualified Publisher/Contractor 

The persons involved in the review of this task/activity included three instructors, one 
contractor, and four project administrative staff members. Following are the responses 
of the reviewers for each question used for the evaluation of this task/ activity. 



12 



ERLC 



00 



Questions 



Process Stage 

Percent of Reviewers 
Responding 



1. Were capability and bid evaluation criteria Yes 100 No , 
established? 

2. Was a bid proposal format developed and Yes 100 No , 
used? 

3< Were state guidelines and bid procedures Yes 100 No 

followed? 

4. Was a bid proposal evaluation form Yes 100 No 
developed and used? 

5. Was the response to the bid proposals done Yes 100 No 
in a timely manner? 



Product Stag e 
Task/Activity: Development of Interactive Video Disc (IVD) 

The persons involved in the review of this task/activity included three instmctors, three 
project advisory committee members, three project administrative staff members, and 
one contractor. Following are the responses of the reviewers for each question used for 
the evaluation of this task/activity. 

Percent of 
Reviewers 

Questions Responding 

1. Was the IVD developed as proposed? Yes 90 No JO^ 

2. Did the IVD meet contract specifications? Yes ,80_ No ,20. 

3. Does the IVD provide for the development of all skills Yes 80 No 10 
identified by the literacy audit and the task analysis? 
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Task/Activity: Interactive Videodisc Training 

The persons involved in the review of this task/activity included three instructors and 
one project administrative staff member. Following are the responses of the reviewers 
for each question used for the evaluation of this task/activity. 

Percent of 
Reviewers 

Questions Responding 

1. Did the IVD training proceed as scheduled? Yes 75 No 25 

2. Did the IVD training meet the instructional objectives as Yes 100 No 

determined by the literacy audit? 

3. Was the IVD training evaluated by students and staff? Yes 100 No 

4. Was the time established for IVD training sufficient? Yes 100 No 

5. Were sufficient resources needed to supplement the IVD Yes 100 No 

training made available? 



Task/Activity: Project Management Plan 

The persons involved in the review of this task/activity included three instructors, one 
partner, four project administrative staff members, one contractor, and the independent 
evaluator. Following are the responses of the reviewers for each question used for the 
evaluation of this task/activity. 

Percent of 
Reviewers 

Questions Responding 

L Were the milestones and timelines met as established? Yes 89 No 11 

2. Were revisions made as the result of unforeseen barriers Yes 100 No 

that were not under the control of project management? 



Task/Activity: Program Design and Planning 
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The persons involved in the review of this task/activity included two instructors, four 
project advisory committee members, and two project administrative staff members. 
Following are the responses of the reviewers for each question used for the evaluation of 
this task/activity. 



Questions 

1. Were the proposed resources used in the design? 

2. Were additional resources identified? 

3. Were the total resources sufficient? 

4. Were provisions made in the design to assure the 
curricula would be competency-based? 

5. Were the proposed methods of delivery used in the 
design? 



Percent of 
Reviewers 
Responding 

Yes 100 No, 

Yes 100 No 

Yes 100 No 

Yes 100 No 

Yes 100 No 



Task/ Activity: Student Achievement 

The persons involved in the review of this task/activity consisted of one partner. 
Following are the responses of the reviewers for each question used for the evaluation of 
this task/activity. 

Percent of 
Reviewers 

Questions Responding 

1. Rate the increase in the productivity of all employees as a 
group after training by checking one of the statements 
below which best indicates your rating of increased 
productivity: 

1. No increase in productivity 

2. Some increase in productivity 

3. Moderate increase in productivity 100 
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4. Above average increase in productivity 

5, High increase in productivity 

2. Was there improvement in communications and 
mathematical skills of employees? 

3. Was there improvement in the workplace attitude of 
employees? 

4. Was there a decrease in absenteeism of employees? 

5. Would you recommend that other employees needing 
literacy improvement be provided this training? 

Task/Activity: Students with IVD Training 

The persons involved in the review of this task/activity included five partners. Following 
are the responses of the reviewers for each question used for the evaluation of this 
task/activity. 

Percent of 
Reviewers 

Questions Responding 

1. Rate the increase in the productivity of all employees as a 
group after training by checking one of the statements 
below which best indicates your rating of increased 
productivity: 

1. No increase in productivity 40 

2. Some increase in productivity 

3. Moderate increase in productivity 40 

4. Above average increase in productivity 20 

5. High increase in productivity 



Yes 100 No 

Yes 100 No 



Yes 100 No 
Yes 100 No 
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2. Was there improvement in communications and mathematical Yes lOQ No 

skills of employees? 

3. Was there improvement in the workplace attitude of Yes IQQ No 

employees? 

4. Was there a decrease in absenteeism of employees? Yes .20. No 80 

5. Would you recommend that other employees needing literacy Yes IQQ No 

improvement be provided this training? 

6. Do you feel that students (employees) participating in IVD Yes IQO No 

training made greater progress in training using traditional 

methods of teaching? 



Task/ Activity: Evaluation Pbn IVD Field Test 



[The persons involved in the review of this task/activity included instructors, project advisory 
committee members, project administrative staff and the indq)endent evaluator. Following 
are the responses of the reviewers for each question used for the evaluation of this 
task/activity.] 

Percent of Reviewers 
Responding 

Questions 

1. Was the field test conducted as proposed? Yes 32 No 67 

2. Were the evaluations of the IVD by the employees (students) Yes 100 No 

positive? 

3. Were the positive evaluations of IVD by students at the field Yes 100 No 

test sites documented? 



4. Were the assessments of IVD made by the adult education Yes iOQ No 

specialists at the field test sites positive? 

5. Did Service Master supervisors at each site feel that the IVD 

would be appUcable for their staff? Yes IQQ No_ 
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6. Did the IVD meet the specific objectives of the instructional Yes 100 No_ 
program? 

7. Did the training meet the intent of the grant? Yes 100 No^ 

8. Did the training meet the training needs identified by the 

literacy audit? Yes 100 No^ 
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Summary of Evaluation Findings 

The summary of the evaluation findings are reported by task/activity in relation to 
the project objective for which the task/activity was performed. 



Objective 1: "Conduct a thorough needs assessment/literacy audit on which to base 
the design of the program." 

Task/Activity: Selection of Literacy Tests 

Five persons were selected to respond to the questions used in the evaluation of this 
task. All five of the respondents agreed that this task was performed as proposed. 
Task/ A ctivity: Literacy A udit 

Thirty-six persons which included twenty-three students responded to the four 
evaluation question s for this task/activity. All thirty-six respondents agreed that the 
literacy' audit v/as conducted as proposed and achieved the purposes for which the 
literacy audit was performed. 

Objective 2: "Conduct Phase I Training" 
Ta'ik/A ctivity: Task A nalysis 

Two instructors, two project administrative staff, and the independent evaluator 
reviewed both the process and products of the task analysis. All reviewers agreed that 
the task analysis was conducted as proposed and the expected results were produced. 

19 



ERLC 



Pre and post tests using ABLE were administered at the Waco site and at the one field 
test site. The results of the pre and post test for the Waco site are presented in table 1. The 
average gain for the class between the pre and post test at the Waco site was eight percent. 
Table 2 shows the results of the evaluation of the training made by students at the Waco site. 
Table 3 shows the results of the pre and post tests used at the Ivey Tech field test site. The 
average gain for the class at the Ivey Tech site was 7.7 percent. The results of the students 
evaluation of the instruction at Ivey Tech are presented in Table 4. 
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TABLE 1 

Comparison of pretest and posttest scores made by students on the ABLE test at Waco. 



Student 


Pretest Score 


Posttest Score 


Points Gained 


1 


75 


87 


12 


2 


83 


87 


4 


3 


69 


71 


2 


4 


67 


82 


15 


5 


67 


82 


15 


6 


70 


76 


6 


7 


87 


92 


5 


8 


84 


86 


2 


9 


87 


86 


-1 


10 


82 


91 


9 


11 


64 


80 


16 


12 


82 


83 


1 


13 


83 


88 


5 


14 


70 


73 


3 


15 


60 


89 


29 


16 


80 


85 


5 


17 


70 


85 


15 


18 


68 


77 


9 


19 


88 


94 


6 


20 


91 


86 


-5 


21 


77 


93 


16 


22 


86 


94 


8 


23 


77 


82 


5 


24 


64 


85 


21 


25 


84 


93 


9 


26 


86 


91 


5 


27 


91 

1 


91 


0 



Vlean of points gained: 8.0 
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Table 2 

WACO WORKPLACE EDUCATIONAL SKILLS TRAINING 
Student Program Evaluation 



(1) 


1 


Too little 


The amount of time allowed for this course was: 


15 


About Right 




0 


Too much 


(2) 


1 


Too little 


The amount of work assigned was: 


15 


About Right 




0 


Too much 


(3) 


3 


Not at all 


The coursework applies to my job: 


10 


Somewhat 




3 


Frequently 


(4) 


0 


Never 


I wish to participate in another class: 


8 


Later 




9 


Immediately 




0 


Too early 


The time of day for this class was: 


15 


About right 




1 


Too late 


(6) 


4 


Reading 


I would like more training in: 


5 


Writing 




7 


Math/Physic 




15 


Computer 
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Table 3 



Comparison of pretest and posttest scores made by students on ABLE test at Ivey Tech. 



Student 


Pretest Score 


Posttest Score 


Points Odined 




35 


80 


45 


2 


75 


85 


10 


3 


75 


85 


10 






on 

OU 


•J 


< 




85 


10 


5 


80 


80 


0 


7 


85 


90 


5 


O 

o 


oU 




c 


9 


75 


80 


5 


/ 10 


80 


85 


5 


11 


80 


85 


5 


12 


75 


85 


10 


13 


85 


90 


5 


14 


75 


85 


10 


15 


80 


80 


0 


16 


75 


85 


10 


17 


70 


85 


15 


18 


75 


85 


10 


19 


80 


85 


5 


20 


75 


85 


10 


21 


85 


85^ 


0 


22 


75 


80 


5 


23 


75 


85 


10 


24 


80 


85 


5 


25 


75 


85 


10 


26 


70 


85 





27 


85 


80 


^ 5 



Mean of the Points Gained: 7.78 
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Table 4 
Ivy Tech Pilot Site Results 
"Workplace Educational Skills Training 
Student Program Evaluation 





7 

/ 




1 ne diuouni oi iinic auowea lor inis course was. 


on 
zu 


ADOUt rigni 




1 


Ton Kfiirh 




9 


X oo uciie 


i dmouiii UL wuiK dooigncu wd:>. 




/\DOUi rigni 




4 


Too much 




4 


TSJnt at all 










g 


F*rf*niiPntlv 


V*) 




Never 


I wish to participate in another class: 


18 


Later 




7 


Immediately 


(5) 


0 


Too early 


The time of day for this class was: 


28 


About right 




0 


Too late 


(6) 


2 


Reading 


I would like more training in: 


0 


Writing 




7 


Math/Physics 




16 


Computer 



Task/Activity: Program Design and Planning 

The eight reviewers of this task, which included four members of the project advisory 
committee, all agreed that the criteria for evaluating this task had been achieved. This 
task/activity was also evaluated under the Product stage of the evaluation process. 

Task/Activily: Curriculum Development 

Eleven persons reviewed both the process and product of this task. It was the 
consensus of the reviewers that all seven of the criteria used to measure the achievement 
of this task had been met. 

Task/Activity: Purchase of Instructional Materials 
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Seven persons, including one partner, responded to the questions used in determining 
if this task had been performed as proposed. All of the reviewers responded **yes" to 
questions used in evaluating this task. 

Task/Activity: Student Training (phase I) 

The reviewers of this task/acti\'ity did not agree that all of the criteria used in 
evaluating this task/activity had been met at 100 percent. One reviewer felt that based 
on his/her definition of "training plan" that individual training plans had not been 
developed. One reviewer felt that the training did not proceed as scheduled. Two of the 
respondents did not agree that r.tudent release time was sufficient. Comment made by 
these two reviewers were: "The students should have been allowed to attend training 
without having to donate their personal time i.e., lunch hour," and "Time was supposed 
to be given by Servicemaster, not 1 for 1." The reviewers agreed that a]l of the other 
criteria used in evaluating this task had been met. 

The evaluations of the students for student training at the Waco site were positive. 
Fifty-three percent of the students responding to the questionnaire indicated that they 
desired to participate in additional training immediately. Although all of the students 
responding expressed a desire to participate in future training, two students did not feel 
that the coursework applied to their jobs. 

Comments made by the responding students included: 

*i really enjoyed the studies that we had. Thanks for all of your help." 

"I greatly enjoyed the classes and our teacher was an absolute delight. I would love 
to take more classes. Thank you." 
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"I would like more time with computer operations." 

"I enjoyed the training very much. My instructor, Melinda Ziegler, made me very 
comfortable in the class. oom." 

"I feel that the classes would be valuable to anyone willing to take them." 
Task/Activity: Selection of a Qualified/Publisher Contractor 

All of the individuals evaluating this task agreed that all of the evaluation criteria had 
been met. 

Task/Activity: Development of Video Disc (IVD) 

This task was evaluated by six project staff members, three project advisory 
committee members, and one staff member of the contractor. The evaluations of this 
task were performed prior to June 10, 1992. The evaluators of this task did not agree 
that the interactive video disc for the curriculum had been developed as proposed and 
within the timeframe established. Following are the comments made by evaluators who 
did not feel the task had been achieved as proposed: 

"IVD not the only thing to address deficiencies-other methods used to address same 
needs identified by the literacy audit." 

"At this time modifications to the IVD have not been received." 

"IVD has yet to be completed." 

Task/Activity: Interactive Videodisc Training 

Three instructors and one project administration staff member reviewed the 
interactive videodisc training. Overall the reviewers agreed that the training was 
conducted as proposed. One evaluator did agree that the trainmg proceeded as 
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scheduled. The comment of this evaluator was: "Due to late start training was delayed 
but progressed at projected rate." 

Task/Activity: Program Design and Planning 

Part of the program design and training evaluation data was presented under the 
evaluation stage of installation. The remaining part of the evaluation of this task/activity 
was performed under the product stage. The evaluators of the product stage for 
Program Design and Planning agreed that the product criteria had been met. 

Task/Activity: Student Achievement 

Student achievement was evaluated on-site by five Service Master supervisors of the 
student-employees and by the use of ABLE pretest and posttest. 

Student achievement of Phase I training was evaluated by five supervisors of Service 
Master. The evaluator felt that there was an above average increase in productivity as a 
result of the Phase I training. The evaluator observed that the supervisor's ratings were 
positive and would reconmiend that other employees needing literacy improvement be 
provided with the training. 

Five of the supervisors evaluated student achievement following the IVD training. 
One of the supervisors felt that the student-employees as a group achieved an above 
average increase in productivity as a result of the training. 

Two felt that there was not increase in training and two felt that the student- 
employees had made a moderate increase in productivity. One of the supervisors felt 
that the training had decreased the absenteeism of the student-employees. 
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All of the supervisors agreed that there was in improvement of the student-employees 
in the areas of communications and mathematical skills, and workplace attitude as a 
itesult of the training. 

All of the supervisors agreed that they would recommend the training for other 
employees needing literacy improvement. All of the supervisors agreed that student- 
employees participating in the IVD training made greater progress than those in training 
using traditional methods of teaching. 

Objective 3: "Addressing the Computational Literacy Needs at the Management 
Level" 

This objective was removed from the project during contract negotiations with the 
Department of Education. 

Objective 4: "Conduct a complete evaluation of the project and, if results are as 
expected, prepare for dissemination of the program throughout the corporation and 
other educational institutions interested in serving SM workers nationally an 
internationally." 

■ . - 'i' 

Task/ Activity: Oversight Committee 

This task/activity was evaluated by the independent evaluator. It was found that the 
oversight committee, which is called the Project Transformation Committee, served the 
project as proposed. The independent evaluator attended all of the Oversight 
Committee meetings. 
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Task/Activity: Third-Party Evaluator 

The third-party evaluator, referred to as the independent evaluator, was obtained as 
proposed- The third party evaluator assisted the project staff in developing and 
implementing a project evaluation plan in accordance with the specifications agreed to in 
the contract negotiations with the Department of Education. The evaluation plan 
developed to guide in the evaluation of the project was reviewed and approved by the 
Oversight Committee. 

Task/Activity: Field Test of the Interactive Videodisc 
It was proposed that the interactive videodisc training would be field tested a ten sites 
selected by the business/industry Partner of the project. Due to an unforeseen change in 
the administration of the Partner, the field tests were not made at the ten sites. The 
field test was made at one site long after the scheduled dates for the field tests. 
The findings from the field test were similar to those of training conducted using the 
interactive videodisc on the Waco campus. The Partner's evaluations of the interactive 
videodisc field test were all positive except for the question "Was the field lest 
conducted as proposed?" 

Based on the information provided to the third party evaluator, the Texas State 
Technical College at Waco, Texas was not as fault for this task not being achieved as 
proposed. 

Task/Activity: Management Plan 

The project staff developed a management using the Program Evaluation Review 
Technique and Management by Objective following grant approved by the U.S. 
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Department of Education. The evaluation plan or the project was developed to parallel 
the management plan. 

The management plan used for this project was developed by analyzing each objective 
for identifying and scheduling major tasks to be performed. Activities were identified 
and listed for each of the major tasks. This structure of the management plan provided 
for the interaction of organizational structure, performance of individuals, and a 
monitoring procedure of activity and quality control using a projected time frame for the 
completion of each task and objective. 

With the exception of the proposed field test of the interactive video disc, adjustments 
were made to the management plan without detriment to the project. 
It was observed that the field tests were not conducted as proposed due to administrative 
changes of the Partner, Service Masters Inc. Service Masters was only able to arrange 
for a field test at one site. 



30 



Conclusion 



Based on the observations made by the independent evaluator of the evaluation data 
collected, the project has achieved the goal of the demonstration project by developing, 
testing by implementation^ and proven by student achievement a model program of 
applied Workplace Literacy. This model or demonstration program can be used 
throughout the nation by Service Master, Inc. and adapted or replicated for use by other 
businesses and industries providing similar services and products. 

It is my conclusion that the project has achieved all of the objectives and tasks of the 
proposed project. It is also concluded, that the use of the interactive for workplace 
literacy training is more effective than are traditional methods and educational 
technologie:: 




Bill E. Lovelace, Independent Evaluator 



December 28, 1992 
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